A study in rule-specific issue categorization for e-rulemaking

We address the e-rulemaking problem of categorizing public comments according to the issues that they address. In contrast to previous text categorization research in e-rulemaking [5, 6], and in an attempt to more closely duplicate the comment analysis process in federal agencies, we employ a set of rule-specific categories, each of which corresponds to a significant issue raised in the comments. We describe the creation of a corpus to support this text categorization task and report interannotator agreement results for a group of six annotators. We outline those features of the task and of the e-rulemaking context that engender both a non-traditional text categorization corpus and a correspondingly difficult machine learning problem. Finally, we investigate the application of standard and hierarchical text categorization techniques to the e-rulemaking data sets and find that automatic categorization methods show promise as a means of reducing the manual labor required to analyze large comment sets: the automatic annotation methods approach the performance of human annotators for both flat and hierarchical issue categorization. 1. BACKGROUND AND INTRODUCTION Every year federal agencies publish in the Federal Register [on-line version: http://www.gpoaccess.gov/fr/index.html] several thousand documents on which they seek public comment. Most of these are proposed rules in areas including: environmental protection; agriculture standards; drug, workplace, and consumer safety; import and export controls; air, highway, and water-based transportation safety; communications; and various federal grant and aid programs. Federal statutes, especially the Administrative Procedure Act [5 U.S. C. §§551 et seq.], generally require such regulations to go through this “notice and comment” process before they can become final and binding on the public. In addition, agencies may request comments on a category of documents known as “guidance.” These documents, which often closely resemble proposed rules in form, have different names but share the characteristic that they are advice, or warning, about how the agency will exercise its power or interpret the law, rather than binding rules themselves. Sometimes, the agency is legally required to seek public comment before it finalizes guidance; other times, it simply chooses to do so. Finally, there is a third, miscellaneous category of documents in the Federal Register on which comments are solicited. These include such things as draft statements of environmental or other impacts of proposed agency action. Again, the agency may be required by one of its statutes to allow the public to comment, or it may be doing so as a matter of policy. Until the 1990s, all comments came to the agency in hard copy— through hand delivery, conventional, or express mail. As electronic transmission and then the Internet became more generally available, agencies began to receive comments first by fax and then by e-mail. Most recently, agencies have provided web portals for comment submission. Although a few agency-specific sites remain, most have been superseded by a central portal, www.regulations.gov, which now provides access to all agencies’ proposed rules and guidance, as well as to some of the documents in the third, miscellaneous category. Comments can be submitted to all agencies through this portal. (Commentors can continue to use the older submission methods as well.) Transfer of the notice-and-comment process to the web — and, more broadly, the use of information technology to support any step in the rulemaking process — is known as electronic rulemaking (e-rulemaking). At the close of the public comment period (typically, 30–60 days) the comments must be reviewed to determine what issues they contain. The comment process is not a vote; its purpose is not to tally the commentors’ preferences for or against the proposal. By law, agencies must act based on factors, and to further objectives, specified by their authoThese include such widely used names as “statements of policy” and “interpretive rule,” as well as more agencyspecific names as “circular” and “bulletin.” The Proceedings of the 9th Annual International Digital Government Research Conference